1. Field of the Invention
The present invention relates generally to the field of video conferencing, and pertains particularly to systems and methods for augmenting user perspective in multi-perspective multi-point tele-immersive environments.
2. Discussion of the State of the Art
In the field of video conferencing and more particularly, E learning tele-immersive environments involving a local educational source and a classroom that may incorporate remote classrooms of students geographically distant from the local classroom, by establishing a classroom specific architecture of video capture devices or cameras and video display devices supported on a local area network (LAN) that has access to a carrier wide area network (WAN).
The inventors are aware of a system described in this specification, wherein software running on at least one LAN-connected server in each connected classroom manages multiple perspective views of the education source (teacher, whiteboard, media presentation) and multiple perspective views of the local students and the students of each of the connected remote classrooms. The system known to the inventor maps the appropriate camera feeds of the remote students to display for the teacher or lecturer, and the appropriate camera feeds of the teacher or lecturer to the remote students, based on algorithmic computation of the best gaze/feed angle based on positions of the teacher and students dependent upon gesture recognition of gestures by the teacher and recognition of gestures by the students as to one of several modes of state activity defined for the environment.
Symmetrical arrangement of video cameras and video displays and arranged and oriented student areas for the remote classrooms, provide the base angles for viewing and views afforded to the teacher and students are optimized in real time by making corrections in gaze. More particularly, gaze alignment correction as described utilizes a unique observer-dependent vector (ODV) system to calculate the gaze correction factor before transforming the 3D objects (teacher and students) into 2D projection space.
The afore-mentioned method of the system includes using a coordinate system for locating and defining the objects in a 3D environment, a set of vector definitions and the behavior of vectors in response to changes in environment. The method is used to model human gaze in order to calculate the gaze correction factor that will enable the local and remote participants to view each other as if they were physically present in the same physical location.
The head movements (and other gestures) of all the participants are tracked and analyzed utilizing the unique ODV system to calculate the gaze correction factor, which is then used to find the exact 2D projection of the 3D object that is projected on any given display device. In practice of the invention, each participant sees a different perspective of other participants which preserves the 3D behavior of objects on a 2D display device.
The system as previously known is somewhat limited in upwards scalability where larger numbers of remote classrooms may be involved. To maintain efficiency and granularity of imagery, more video capture devices and more display devices are required, inflating equipment costs. Furthermore, having more cameras and perspectives cause more processing in real time and require greater bandwidth in transmission over a carrier network.
Therefore, what is clearly needed is a network-based multi-point, multi-perspective, tele-immersive video conferencing system that incorporates augmented reality (AR) and three-dimensional (3D) reconstruction techniques to reduce the complexity of required equipment and to save networking resources.